from sklearn_benchmarks.reporting.hp_match import HpMatchReporting
reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=True)
reporting.make_report()
We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup
time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.
KNeighborsClassifier_brute_force¶onnx (1.10.1) vs. scikit-learn (1.0.dev0)
All estimators share the following parameters: algorithm=brute.
| estimator | function | diff_accuracy_scores | n_samples_train | n_samples | n_features | mean_duration_sklearn | std_duration_sklearn | n_iter | iteration_throughput | latency | algorithm | n_jobs | n_neighbors | accuracy_score_sklearn | mean_duration_onnx | std_duration_onnx | accuracy_score_onnx | speedup | std_speedup | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | KNeighborsClassifier_brute_force | predict | 0.337 | 100000 | 1000 | 100 | 1.994884 | 0.172649 | NaN | 0.000401 | 0.001995 | brute | -1 | 1 | 0.663 | 0.354387 | 0.005473 | 1.000 | 5.629111 | 5.629782 |
| 2 | KNeighborsClassifier_brute_force | predict | 0.243 | 100000 | 1 | 100 | 0.022639 | 0.002272 | NaN | 0.000035 | 0.022639 | brute | -1 | 1 | 1.000 | 18.982136 | 0.359374 | 0.757 | 0.001193 | 0.001193 |
| 4 | KNeighborsClassifier_brute_force | predict | 0.125 | 100000 | 1000 | 100 | 2.951273 | 0.047169 | NaN | 0.000271 | 0.002951 | brute | -1 | 5 | 0.757 | 19.146237 | 0.042717 | 0.882 | 0.154144 | 0.154144 |
| 7 | KNeighborsClassifier_brute_force | predict | 0.118 | 100000 | 1000 | 100 | 2.199261 | 0.025263 | NaN | 0.000364 | 0.002199 | brute | 1 | 100 | 0.882 | 0.354428 | 0.013963 | 1.000 | 6.205100 | 6.209914 |
| 8 | KNeighborsClassifier_brute_force | predict | 0.243 | 100000 | 1 | 100 | 0.020690 | 0.001177 | NaN | 0.000039 | 0.020690 | brute | 1 | 100 | 1.000 | 19.135550 | 0.244916 | 0.757 | 0.001081 | 0.001081 |
| 10 | KNeighborsClassifier_brute_force | predict | 0.219 | 100000 | 1000 | 100 | 2.924734 | 0.059220 | NaN | 0.000274 | 0.002925 | brute | -1 | 100 | 0.882 | 19.682121 | 0.255039 | 0.663 | 0.148599 | 0.148611 |
| 13 | KNeighborsClassifier_brute_force | predict | 0.243 | 100000 | 1000 | 100 | 2.188334 | 0.040193 | NaN | 0.000366 | 0.002188 | brute | 1 | 5 | 0.757 | 0.272164 | 0.006575 | 1.000 | 8.040485 | 8.042831 |
| 14 | KNeighborsClassifier_brute_force | predict | 0.078 | 100000 | 1 | 100 | 0.020871 | 0.001000 | NaN | 0.000038 | 0.020871 | brute | 1 | 5 | 1.000 | 4.029837 | 0.085303 | 0.922 | 0.005179 | 0.005180 |
| 16 | KNeighborsClassifier_brute_force | predict | 0.266 | 100000 | 1000 | 100 | 1.223682 | 0.020244 | NaN | 0.000654 | 0.001224 | brute | 1 | 1 | 0.663 | 4.128764 | 0.072914 | 0.929 | 0.296380 | 0.296426 |
| 19 | KNeighborsClassifier_brute_force | predict | 0.104 | 100000 | 1000 | 2 | 1.747191 | 0.032236 | NaN | 0.000009 | 0.001747 | brute | -1 | 1 | 0.896 | 0.268770 | 0.006979 | 1.000 | 6.500682 | 6.502873 |
| 20 | KNeighborsClassifier_brute_force | predict | 0.078 | 100000 | 1 | 2 | 0.005002 | 0.002656 | NaN | 0.000003 | 0.005002 | brute | -1 | 1 | 1.000 | 4.012612 | 0.083597 | 0.922 | 0.001247 | 0.001247 |
| 22 | KNeighborsClassifier_brute_force | predict | 0.026 | 100000 | 1000 | 2 | 2.764815 | 0.041363 | NaN | 0.000006 | 0.002765 | brute | -1 | 5 | 0.922 | 3.994434 | 0.046559 | 0.896 | 0.692167 | 0.692214 |
predict
| estimator | function | n_samples_train | n_samples | n_features | mean_duration_sklearn | std_duration_sklearn | iteration_throughput | latency | n_jobs | n_neighbors | accuracy_score_sklearn | mean_duration_onnx | std_duration_onnx | accuracy_score_onnx | speedup | std_speedup | sklearn_profiling | onnx_profiling | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 1.995 | 0.173 | 0.000 | 0.002 | -1 | 1 | 0.663 | 0.354 | 0.005 | 1.000 | 5.629 | 5.630 | See | See |
| 2 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 0.023 | 0.002 | 0.000 | 0.023 | -1 | 1 | 1.000 | 18.982 | 0.359 | 0.757 | 0.001 | 0.001 | See | See |
| 4 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 2.951 | 0.047 | 0.000 | 0.003 | -1 | 5 | 0.757 | 19.146 | 0.043 | 0.882 | 0.154 | 0.154 | See | See |
| 5 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 0.025 | 0.003 | 0.000 | 0.025 | -1 | 5 | 1.000 | 0.353 | 0.013 | 1.000 | 0.072 | 0.072 | See | See |
| 7 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 2.199 | 0.025 | 0.000 | 0.002 | 1 | 100 | 0.882 | 0.354 | 0.014 | 1.000 | 6.205 | 6.210 | See | See |
| 8 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 0.021 | 0.001 | 0.000 | 0.021 | 1 | 100 | 1.000 | 19.136 | 0.245 | 0.757 | 0.001 | 0.001 | See | See |
| 10 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 2.925 | 0.059 | 0.000 | 0.003 | -1 | 100 | 0.882 | 19.682 | 0.255 | 0.663 | 0.149 | 0.149 | See | See |
| 11 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 0.028 | 0.005 | 0.000 | 0.028 | -1 | 100 | 1.000 | 0.363 | 0.013 | 1.000 | 0.076 | 0.076 | See | See |
| 13 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 2.188 | 0.040 | 0.000 | 0.002 | 1 | 5 | 0.757 | 0.272 | 0.007 | 1.000 | 8.040 | 8.043 | See | See |
| 14 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 0.021 | 0.001 | 0.000 | 0.021 | 1 | 5 | 1.000 | 4.030 | 0.085 | 0.922 | 0.005 | 0.005 | See | See |
| 16 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 1.224 | 0.020 | 0.001 | 0.001 | 1 | 1 | 0.663 | 4.129 | 0.073 | 0.929 | 0.296 | 0.296 | See | See |
| 17 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 0.021 | 0.002 | 0.000 | 0.021 | 1 | 1 | 1.000 | 0.274 | 0.007 | 1.000 | 0.075 | 0.075 | See | See |
| 19 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 2 | 1.747 | 0.032 | 0.000 | 0.002 | -1 | 1 | 0.896 | 0.269 | 0.007 | 1.000 | 6.501 | 6.503 | See | See |
| 20 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 2 | 0.005 | 0.003 | 0.000 | 0.005 | -1 | 1 | 1.000 | 4.013 | 0.084 | 0.922 | 0.001 | 0.001 | See | See |
| 22 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 2 | 2.765 | 0.041 | 0.000 | 0.003 | -1 | 5 | 0.922 | 3.994 | 0.047 | 0.896 | 0.692 | 0.692 | See | See |
| 23 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 2 | 0.007 | 0.005 | 0.000 | 0.007 | -1 | 5 | 1.000 | 0.283 | 0.007 | 1.000 | 0.026 | 0.026 | See | See |
KNeighborsClassifier_kd_tree¶onnx (1.10.1) vs. scikit-learn (1.0.dev0)
All estimators share the following parameters: algorithm=kd_tree.
| estimator | function | diff_accuracy_scores | n_samples_train | n_samples | n_features | mean_duration_sklearn | std_duration_sklearn | n_iter | iteration_throughput | latency | algorithm | n_jobs | n_neighbors | accuracy_score_sklearn | mean_duration_onnx | std_duration_onnx | accuracy_score_onnx | speedup | std_speedup | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | KNeighborsClassifier_kd_tree | predict | 0.071 | 1000000 | 1000 | 10 | 0.891720 | 1.171892 | NaN | 0.000090 | 0.000892 | kd_tree | -1 | 1 | 0.929 | 2.997673 | 0.262560 | 1.000 | 0.297471 | 0.298609 |
| 2 | KNeighborsClassifier_kd_tree | predict | 0.054 | 1000000 | 1 | 10 | 0.002790 | 0.000454 | NaN | 0.000029 | 0.002790 | kd_tree | -1 | 1 | 1.000 | 127.169801 | 0.000000 | 0.946 | 0.000022 | 0.000022 |
| 4 | KNeighborsClassifier_kd_tree | predict | 0.005 | 1000000 | 1000 | 10 | 1.141847 | 0.397180 | NaN | 0.000070 | 0.001142 | kd_tree | -1 | 5 | 0.946 | 128.199489 | 0.000000 | 0.951 | 0.008907 | 0.008907 |
| 7 | KNeighborsClassifier_kd_tree | predict | 0.049 | 1000000 | 1000 | 10 | 5.498338 | 0.533157 | NaN | 0.000015 | 0.005498 | kd_tree | 1 | 100 | 0.951 | 3.135518 | 0.319834 | 1.000 | 1.753566 | 1.762665 |
| 8 | KNeighborsClassifier_kd_tree | predict | 0.054 | 1000000 | 1 | 10 | 0.003219 | 0.000686 | NaN | 0.000025 | 0.003219 | kd_tree | 1 | 100 | 1.000 | 127.138928 | 0.000000 | 0.946 | 0.000025 | 0.000025 |
| 10 | KNeighborsClassifier_kd_tree | predict | 0.022 | 1000000 | 1000 | 10 | 3.021899 | 0.291408 | NaN | 0.000026 | 0.003022 | kd_tree | -1 | 100 | 0.951 | 126.774528 | 0.000000 | 0.929 | 0.023837 | 0.023837 |
| 13 | KNeighborsClassifier_kd_tree | predict | 0.054 | 1000000 | 1000 | 10 | 1.672402 | 0.245720 | NaN | 0.000048 | 0.001672 | kd_tree | 1 | 5 | 0.946 | 0.005896 | 0.000206 | 1.000 | 283.629602 | 283.801915 |
| 14 | KNeighborsClassifier_kd_tree | predict | 0.089 | 1000000 | 1 | 10 | 0.001587 | 0.000485 | NaN | 0.000050 | 0.001587 | kd_tree | 1 | 5 | 1.000 | 0.044325 | 0.003629 | 0.911 | 0.035812 | 0.035932 |
| 16 | KNeighborsClassifier_kd_tree | predict | 0.035 | 1000000 | 1000 | 10 | 0.904323 | 0.193756 | NaN | 0.000088 | 0.000904 | kd_tree | 1 | 1 | 0.929 | 0.067247 | 0.002510 | 0.894 | 13.447729 | 13.457091 |
| 19 | KNeighborsClassifier_kd_tree | predict | 0.109 | 1000 | 1000 | 2 | 0.033206 | 0.020860 | NaN | 0.000482 | 0.000033 | kd_tree | -1 | 1 | 0.891 | 0.005599 | 0.000693 | 1.000 | 5.931034 | 5.976345 |
| 20 | KNeighborsClassifier_kd_tree | predict | 0.089 | 1000 | 1 | 2 | 0.003056 | 0.000374 | NaN | 0.000005 | 0.003056 | kd_tree | -1 | 1 | 1.000 | 0.042173 | 0.002047 | 0.911 | 0.072468 | 0.072554 |
| 22 | KNeighborsClassifier_kd_tree | predict | 0.020 | 1000 | 1000 | 2 | 0.024285 | 0.001525 | NaN | 0.000659 | 0.000024 | kd_tree | -1 | 5 | 0.911 | 0.041578 | 0.002049 | 0.891 | 0.584086 | 0.584795 |
predict
| estimator | function | n_samples_train | n_samples | n_features | mean_duration_sklearn | std_duration_sklearn | iteration_throughput | latency | n_jobs | n_neighbors | accuracy_score_sklearn | mean_duration_onnx | std_duration_onnx | accuracy_score_onnx | speedup | std_speedup | sklearn_profiling | onnx_profiling | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 0.892 | 1.172 | 0.000 | 0.001 | -1 | 1 | 0.929 | 2.998 | 0.263 | 1.000 | 0.297 | 0.299 | See | See |
| 2 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 0.003 | 0.000 | 0.000 | 0.003 | -1 | 1 | 1.000 | 127.170 | 0.000 | 0.946 | 0.000 | 0.000 | See | See |
| 4 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 1.142 | 0.397 | 0.000 | 0.001 | -1 | 5 | 0.946 | 128.199 | 0.000 | 0.951 | 0.009 | 0.009 | See | See |
| 5 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 0.003 | 0.001 | 0.000 | 0.003 | -1 | 5 | 1.000 | 2.960 | 0.217 | 1.000 | 0.001 | 0.001 | See | See |
| 7 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 5.498 | 0.533 | 0.000 | 0.005 | 1 | 100 | 0.951 | 3.136 | 0.320 | 1.000 | 1.754 | 1.763 | See | See |
| 8 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 0.003 | 0.001 | 0.000 | 0.003 | 1 | 100 | 1.000 | 127.139 | 0.000 | 0.946 | 0.000 | 0.000 | See | See |
| 10 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 3.022 | 0.291 | 0.000 | 0.003 | -1 | 100 | 0.951 | 126.775 | 0.000 | 0.929 | 0.024 | 0.024 | See | See |
| 11 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 0.006 | 0.002 | 0.000 | 0.006 | -1 | 100 | 1.000 | 2.964 | 0.186 | 1.000 | 0.002 | 0.002 | See | See |
| 13 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 1.672 | 0.246 | 0.000 | 0.002 | 1 | 5 | 0.946 | 0.006 | 0.000 | 1.000 | 283.630 | 283.802 | See | See |
| 14 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 0.002 | 0.000 | 0.000 | 0.002 | 1 | 5 | 1.000 | 0.044 | 0.004 | 0.911 | 0.036 | 0.036 | See | See |
| 16 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 0.904 | 0.194 | 0.000 | 0.001 | 1 | 1 | 0.929 | 0.067 | 0.003 | 0.894 | 13.448 | 13.457 | See | See |
| 17 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 0.001 | 0.000 | 0.000 | 0.001 | 1 | 1 | 1.000 | 0.006 | 0.000 | 1.000 | 0.188 | 0.188 | See | See |
| 19 | KNeighborsClassifier_kd_tree | predict | 1000 | 1000 | 2 | 0.033 | 0.021 | 0.000 | 0.000 | -1 | 1 | 0.891 | 0.006 | 0.001 | 1.000 | 5.931 | 5.976 | See | See |
| 20 | KNeighborsClassifier_kd_tree | predict | 1000 | 1 | 2 | 0.003 | 0.000 | 0.000 | 0.003 | -1 | 1 | 1.000 | 0.042 | 0.002 | 0.911 | 0.072 | 0.073 | See | See |
| 22 | KNeighborsClassifier_kd_tree | predict | 1000 | 1000 | 2 | 0.024 | 0.002 | 0.001 | 0.000 | -1 | 5 | 0.911 | 0.042 | 0.002 | 0.891 | 0.584 | 0.585 | See | See |
| 23 | KNeighborsClassifier_kd_tree | predict | 1000 | 1 | 2 | 0.002 | 0.000 | 0.000 | 0.002 | -1 | 5 | 1.000 | 0.006 | 0.000 | 1.000 | 0.377 | 0.378 | See | See |
HistGradientBoostingClassifier_best¶onnx (1.10.1) vs. scikit-learn (1.0.dev0)
All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.
| estimator | function | diff_accuracy_scores | n_samples_train | n_samples | n_features | mean_duration_sklearn | std_duration_sklearn | n_iter | iteration_throughput | ... | max_iter | max_leaf_nodes | min_samples_leaf | n_iter_no_change | accuracy_score_sklearn | mean_duration_onnx | std_duration_onnx | accuracy_score_onnx | speedup | std_speedup | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | HistGradientBoostingClassifier_best | predict | 0.176 | 100000 | 1000 | 100 | 0.127203 | 0.005715 | 300 | 0.006289 | ... | 300 | 100 | 100 | 10 | 0.824 | 0.471227 | 0.018186 | 1.0 | 0.269939 | 0.27014 |
1 rows × 23 columns
predict
| estimator | function | n_samples_train | n_samples | n_features | mean_duration_sklearn | std_duration_sklearn | n_iter | iteration_throughput | latency | accuracy_score_sklearn | mean_duration_onnx | std_duration_onnx | accuracy_score_onnx | speedup | std_speedup | sklearn_profiling | onnx_profiling | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | HistGradientBoostingClassifier_best | predict | 100000 | 1000 | 100 | 0.127 | 0.006 | 300 | 0.006 | 0.0 | 0.824 | 0.471 | 0.018 | 1.0 | 0.27 | 0.27 | See | See |